Defer debugpy and psutil imports until they are actually needed - #1543

CI failure will be taken care of by #1544

Both imports were paid for on every kernel startup even though most sessions never debug or ask for usage information.

They still get paid at every kernel startup after this PR. The debugger information is needed at the latest when generating kernel_info_request reply (which every frontend sends). Similar for psutil: init_connection_file → jupyter_client.connect.load_connection_file → _ip_default → localinterfaces._load_ips_psutil.

The import time of import ipykernel does not shrink either as neither debugpy nor psutil was imported on the pure import path (I worked on it in #1223)

A benchmark showing the benefits would help here. Otherwise, there is a number of regressions here:

1. kernel.debugger = X silently kills breakpoints (stock usage unaffected, but subclasses are)
2. IPythonKernel(debugger_class=X) is now silently ignored
3. _get_psutil() publishes the flag before the import (easy to fix though)
4. Debugger init failure is sticky and silent (debug_requests will return None after the first one)

Good point for subclasses, I'll see what I can do.

It's true that I tested only with #1542; that is minimal up to startup point and shut off immediately; and for import only for kernelapp.

The debugger information is needed at the latest when generating kernel_info_request reply (which every frontend sends)

Fair; I did not think about this – but I would not consider kernel_info_reply to be part of startup; I need to check but at least the startup before the first kernel_info_request is faster, I'm assuming the first kernel_info_request is sent only once the kernel is started (I'll verify), so I still think it's a (partial) win, and this should make the time to ready. But I least that's fixable separately and only affect the kernel_info_reply which for me is a separate issue – do we cache the result on disk ? Add a config to force the kernel to reply with 'debugger' always in or never and leave to to large deployment to select 'auto|true|false'

I'm also profiling locally with modified jupyter_client, traitlets and IPython (to defer more stuff from there as well); and yes some thing like psutils are otherwise still in startup path via other packages; but I can try to publish profiles even with modified versions to give an idea of what can be achieved.

I suppose another big things, is are we ok dropping debugger_class traitlet and deprecating it to at least have some way of making debugger import lazy ?

See #1545 for how I think we can handle making kerne_info_reply faster by providing a config option.

Here are the result of

IPYKERNEL_BENCHMARK_STARTUP_SHUTDOWN=1 python tools/importtime_average.py -n 20 -- python -X importtime -m ipykernel  > totlog.log && tuna totlog.log

With #1542 merged in; to stop the kernel immediately and make it measurable. Also requires IPython'm main; traitlet and jupyter_client on latest releases

Also technically you should test with -f fake-connection.json file; we can land a patch in jupyter_client that does not do the _ip_default → localinterfaces._load_ips_psutil when the connection file exists (which is the case most of the time as kernel are launched by frontend that likely can provide IPs in the connection files; but again; different repository. Different optimisation even if that goal is still to avoid importing psutil

for debugger_class I can suggest something like this ipython/traitlets#972, make Type and conditional import to LazyType depending on traitlets version; or pull a minimalistic version that does not allow to check .klass and allow_none=True that i vendor here (which is what I did first):

class _LazyType(Type):  # type:ignore[type-arg]

    def instance_init(self, obj):
        """Deliberately does not resolve; see the class docstring."""
        pass

    def default(self, obj=None):
        self._resolve_classes()
        return super().default(obj)

    def validate(self, obj, value):
        self._resolve_classes()
        return super().validate(obj, value)

Though it becomes brittle as there is no more guard in MetaHasTraits.setup_class that prevent it from importing eagerly.

this now uses a custom local LazyType traitlets, and the flag for the kernel info reply is in another PR.